feat: sparsify snapshots to reduce disk usage by ~67% on Linux and Windows - #12
Merged
Conversation
After writing the HLS snapshot, scan for all-zero 4 KiB pages and punch them with fallocate(PUNCH_HOLE). This makes the file sparse: zeros still read back via mmap but consume no disk blocks. On a typical pyhl snapshot (Python + pandas/numpy warm): Before: 2016 MiB on disk After: 648 MiB on disk (same 2016 MiB apparent size) No load-time overhead — mmap of sparse files is identical to dense. Linux-only; other platforms get a no-op fallback. Signed-off-by: danbugs <danilochiarlone@gmail.com>
Signed-off-by: danbugs <danilochiarlone@gmail.com>
… HANDLE type Signed-off-by: danbugs <danilochiarlone@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
How it works
Linux:
fallocate(PUNCH_HOLE | KEEP_SIZE)on each zero pageWindows:
FSCTL_SET_SPARSE+FSCTL_SET_ZERO_DATAviaDeviceIoControl, with contiguous zero pages coalesced into ranges for fewer syscallsOther platforms: no-op fallback
Changes
host/src/lib.rs:sparsify_snapshot()called aftersave_snapshot()writes the file, with platform-specific implementationshost/src/bin/pyhl.rs:disk_mib()reports actual on-disk size (Linux:statblocks, Windows:GetCompressedFileSizeW) alongside apparent sizehost/Cargo.toml: addslibc,nix(Linux) andwindows-sys(Windows) dependencies for the syscalls